The present invention relates to the field of three dimensional surface reconstruction from a stereoscopy pair of images called stereo pair. Each image of the stereo pair is acquired according to a different viewing angle. The human brain is able to process the stereo pair and mentally reconstruct three dimensions out of it.
Reconstructing a surface in three dimension from a stereo pair is called stereophotogrammetry. Stereophotogrammetry can be based on films or digital images and is extensively being used by cartographers to build topographic maps from aerial images. Through this technology, cartographers are computing level lines enabling the tracing of the relief on maps. Stereophotogrammetry is also used in many applications, including three dimensions reconstruction of industrial objects or skin surface reconstruction for cosmetic or medical applications.
Computers and photography digitizing permitted the development of digital image processing. Amongst the first application of digital image processing is semi-automatized or automatized reconstruction of a three-dimensional surface from a stereo pair. The book “Photogrammetry 1. Fundamentals and Standard Processes”, 4th edition, Karl Kraus and Peter Walrdhäusl, BonnDümmler editing, 1993 is an excellent introduction to the principles of 3D reconstruction in stereophotogrammetry via computers.
Principle of stereophotogrammetry, illustrated by FIG. 1, is to measure accurately the geometric characteristics and to model the optics OA and OB used to acquire the two images of the stereo pair. This step is called calibration. Once the optics are calibrated, by knowing the position PA and PB in each of the two images A and B of the same physical point position P of the surface of the subject S it defines exactly two lines in space whose intersection is the point P on the surface of the subject.
Computer-based stereophotogrammetry is therefore based on two main principles: the knowledge of the geometry of the optics via calibration and the matching of corresponding points in the two images of the stereo pair. A general method to identify corresponding points in both images is to use image cross-correlation, which is maximal when a small window around each point in the images of the stereo pair is compared.
In order to make the image acquisition system more compact for stereophotogrammetry, devices have been developed in order to acquire the two images of the stereo pair by using a single camera body instead of two independent camera bodies.
One way to build such a compact image acquisition system is described in FIG. 2. The device is based on a single camera body 5 and an image splitter constructed with mirrors and enabling the split of the image received by the optical system. For such purpose, one would generally use two external mirrors 1A and 1B, called “secondary mirrors” and spaced at approximately the same distance as human eyes and two internal mirrors 3A and 3B, opposite to the external ones, called “primary mirrors” and which are reflecting the image toward an optical system 4. As natural light conditions are generally not enough for photographic needs, a powerful light system 2 is added on top of the camera.
Such a system, constituted of a camera, a unique flash and an image splitter has been designed by one of the co-authors and is described in the publication: “MAVIS: a non-invasive instrument to measure area and volume of wounds. Measurement of Area and Volume Instrument System”, Plassmann P, Jones T D, Med Eng Phys 1998; 20(5):332-8. FIG. 2 is presenting a device based on an image splitter close to the one used by the authors of this publication.
Another way to build a stereophotogrammetry system using a single body camera is to use two sets of independent lenses for each of the two images to acquire. Such a system, consisting of a camera body, a unique flash and two independent sets of lenses has been designed by the FUJI Company and commercialized under the name of FUJI FinePix 3D W3. FIG. 3 is describing a system based on two independent sets of lenses with the subject S, the camera body 5, the two independent sets of lenses OA and OB and a unique flash 2.
Such systems based on a unique camera body are generally leading to good 3D surface reconstruction except when the surface is reflecting too much the flash light: a phenomenon called “specular reflection”.
A physical surface is reflecting light in two different ways: on one side “diffusion” and on the other side “specular reflection”. Diffusion consists in reflecting received light in a uniform way in space, independently from the incidence angle of the light. Diffusion is specific to mate surfaces. The second way to reflect light is to do it in the same way as a mirror which follows the Snell-Descartes law for light reflection. Such mirror reflection is present in the case of shiny surfaces. Suppose we call the locus of reflection of the light source on a shiny object a “specular spot” as the light source has a given extension and as material property of reflecting surfaces are concentrating more or less the light in the reflection direction, creating a spot rather than a single point.
Specular reflection is creating an issue for 3D surface reconstruction algorithms as the specular spot observed in each of the two optics is shifted between left and right images of the stereo pair in such a way that the method used to find corresponding points is defining a point which is not within the surface to reconstruct but farther away.
In the case of a convex curved surface, the specular spot is reconstructed at a position corresponding to the virtual position of the flash as it is reflected by the shiny surface. It is placed between the surface and up to two times the distance between the surface and the flash depending upon surface curvature. Hence, the point corresponding to the maximal correlation is the mirror image of the flash on the reflecting surface, which is creating an artificial spike oriented backward relative to the surface at the level of the specular spot.
FIG. 4 is describing this geometric phenomenon in the case of a convex shiny object such as a sphere and what happens in the case of a unique camera body 5 with a unique light source 2 and a double optics OA and OB. The light source 2 is creating a specular spot seen at different places of the surface of the sphere by optics OA and optics OB, which is moving the estimated surface reconstructed to P′, that is, backward relative to the subject S surface and not within the surface itself.
A way to reduce the specular effect is to use frosted glass on the flash, which is enlarging and diffusing the specular spot more. However, except when reducing the intensity of the flash by a large amount, the specular spot remains visible and is generally creating a reconstruction artefact.
Another way to reduce the specular effect is to use cross-polarization. Polarization action on light is similar to a “comb”. If one is holding two combs in a parallel way, one on top of each other, one would see through the combs. However, if one is holding these two combs in a perpendicular way, then the light is blocked and one would not be able to see through them. Diffuse reflection is much less sensitive to polarization than specular reflection because specular reflection is reflecting light with exactly the same polarization orientation as incident light. By using a directional polarizing filter on the flash and a polarizing filter with perpendicular orientation on the two optics, one would eliminate most of the specular reflection in the two images of the stereo pair.
Such a technique of cross-polarization has been used by the authors in a previous version of the system. Unfortunately, cross-polarization is removing much of the light and in the example of images of the human skin, they tend to be far from being natural as polarization intensifies the redness of the skin. Another way to avoid such an issue, still with polarization, would be to use two independent flashes and as designed by the authors, consists in using an “inverted” cross polarization between the flashes and the optics, in such a way that the left flash is illuminating the right optics and the right flash is illuminating the left optics. For more details, refer to the English patent application GB 2 468 138 A (UGSC [GB]), Sep. 2010 1 (2010-09-01). This principle of inverted cross polarization has been used in the 3D LifeViz camera as it is presented in HANS SKVARA ET AL: <<Quantification of Skin Lesions with 3D stereovision camera system: validation and clinical applications>>, SKIN RESEARCH AND TECHNOLOGY, vol 19, No 1, 20 février 2013 (2013-02-20), pages e182-e190, XP055139368, ISSN: 0909-752X, DOI:10.1111/j.1600-0846.2012.00625.x, as well as in the case of the 3D LifeViz II camera as described in the report <<Exhibition Watch Report—In-Cosmetics 2013>>, 2013, XP055139703, Paris, page 8 et 9. All these systems are reducing much of the artefacts due to specular spots but are strongly reducing the light in the images because of light attenuation due to the polarization filters which is far from ideal.
A fairly reduced number of other stereophotogrammetry cameras are using two separated flashes not to reduce reconstruction artefacts due to specular reflection but in order to reduce casted shadows on the subject. The device described in the Japanese patent JP 2001 290227 A ((MINOLTA CO LTD) 19 Oct. 2001 (2001-10-19) or the camera <<Holga>> as described in <<The Holga Manual>>, 2011, pages 1-39, XP055140095 are two such systems. These camera systems are not equipped with computation means enabling 3D surface reconstruction and therefore are not adapted to the use of the dual flash to reduce specular reflection.